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ABSTRACT 


Web allows learners to investigate any question to learn with a large number of Web resources. In such investigative 
learning, leaners are expected to investigate the question by navigating Web resources/pages to construct their knowledge 
and decomposing the question into sub-questions. In acquiring skills in such investigative learning, learners need to 
practice with exercise questions. However, it is hard to define correct knowledge to be constructed for the questions since 
Web-based investigative learning could result in diverse knowledge as correct one for the same question. Towards this 
issue, this paper proposes the method with Linked Open Data (LOD) for generating exercise questions, which includes an 
initial question and sub-questions to be decomposed from an initial question. These sub-questions are extracted and 
selected as keywords with LOD and Word2vec. This paper also reports a case study with the generation method. The 
results suggest that it is effective as scaffolding particularly for novice learners. 
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1. INTRODUCTION 


On the Web, learners are expected to investigate any question to learn from a huge number of Web resources. 
Such learning process involves navigating Web resources/pages and constructing wider and deeper 
knowledge from their points of view (Henze and Nejdl, 2001). However, they tend to search a limited 
number of Web resources/pages for investigating a question. As a result, they would construct limited 
knowledge. In elaborately investigating an initial question, learners are expected to deepen and widen the 
question, which requires them to identify related questions to be further investigated during navigation and 
knowledge construction with Web resources (Hill and Hannafin, 1997). This corresponds to decomposing the 
initial question into related ones as sub-questions. 

In addition, Web resources do not usually provide learners with learning scenario implying questions to 
be investigated and their sequence. Learners could not accordingly follow it to construct their knowledge. 
They need to decompose a question into the sub-questions by themselves for elaborate investigation. Such 
question decomposition corresponds to creating a learning scenario, which could play a crucial role in 
self-regulating navigation and knowledge construction process with Web resources (Azevedo and Cromley, 
2004). 

On the other hand, it is not easy for learners to create their own scenario in concurrence with navigation 
and knowledge construction (Zumbach and Mohraz, 2007). In our previous work, we have proposed a model 
of Web-based investigative learning, and developed a tool named interactive Learning Scenario Builder 
(iLSB) as a model-based learning environment (Kashihara and Akiyama, 2017). iLSB provides scaffolding 
for conducting investigative learning process as modeled. We have also confirmed the effectiveness of iLSB 
from a case study whose results suggest iLSB makes learning scenario created by learners wider and deeper. 

In this paper, we address the issue how to improve skills in Web-based investigative learning particularly 
for novice learners who could not create appropriate scenario by themselves. In acquiring cognitive skills, it 
is necessary for learners to practice repeatedly (Azevedo and Cromley, 2004; Azevedo et al., 2015). Our 
approach to this issue is to provide learners with exercise questions for Web-based investigative learning. In 
general, it is necessary to define correct knowledge/scenario to be constructed/created for such exercise 
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questions. However, Web-based investigative learning could result in diverse knowledge and scenario as 
correct ones for the same question. It is accordingly hard to prepare exercise questions with correct results 
(knowledge/scenario to be constructed). 

Toward this issue, we propose a method for generating exercise questions with Linked Open Data (LOD). 
This method extracts and selects keywords representing sub-questions to be decomposed from an initial 
question by means of LOD and Word2vec. It then generates an exercise question, which includes the initial 
and sub-questions. Learners are expected to investigate the exercise question with iLSB and to create a 
learning scenario including the initial and sub-questions. 

In this paper, we also report a case study whose purpose was to evaluate the appropriateness and 
effectiveness of exercise questions generated. The results suggest that these could appropriately and effectively 
function particularly for novice learners. 


2. WEB-BASED INVESTIGATIVE LERNING 


2.1 Difficulties in Web-Based Investigative Learning 


In Web-based investigative learning, it is important for learners to construct deeper and wider knowledge as 
to an initial question by gathering and navigating Web resources, associating the contents learned in the 
resources, and decomposing the initial question into sub-questions in a self-directed way. In addition, most 
Web resources do not have any learning scenario such as table of contents in instructional textbook, which 
implies questions to be investigated and their sequence. Learners accordingly need to create their own 
scenario during navigation and knowledge construction, which results in individual learning. 

On the other hand, it is difficult for learners to concurrently navigate Web resources, construct their 
knowledge, and create their own scenario. This could prevent them from creating more appropriate, wider, and 
deeper learning scenario. In order to address this problem, we have modeled the process of Web-based 
investigative learning, and developed iLSB to provide scaffolds for conducting Web-based investigative 
learning as modeled in our previous work. 


2.2 Model of Web-Based Investigative Learning 


This model consists of three cyclic phases: (a) search for Web resources, (b) navigational learning, and 
(c) question decomposition. In phase (a), learners are expected to use a search engine with a keyword 
representing an initial question to select Web resources suitable for investigating the question. In phase (b), 
they are expected to navigate Web pages in the selected resources and to construct their knowledge by 
extracting keywords representing the contents learned in the pages and associating them. In phase (c), they 
are expected to find out some related sub-questions to be further investigated about the initial question, which 
could be obtained from the keywords extracted in phase (b). They are expected to investigate each 
sub-question in the same way. We assume that these three phases are repeated until they do not decompose 
into sub-questions anymore. 

The question decomposition results in a tree called question tree. In this tree, each node corresponds to a 
question that is represented by a keyword (called q-keyword). In this work, q-keyword is used as a substitute 
for question investigated. This tree includes part of relationships between a question and sub-questions whose 
root represents initial question. This tree also corresponds to learning scenario. 


2.3 Interactive Learning Scenario Builder (iLSB) 


We have developed iLSB as an add-on to Firefox to scaffold investigative learning process as modeled. iLSB 
provides a search engine for selecting Web resources, a keywords repository for knowledge construction, and 
a question tree viewer for question decomposition as scaffolding for each phase of the model. Figure | shows 
the interface of iLSB. 
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Decomposing 
“Greenhouse gas” 
into “Carbon dioxide” 


Figure 1. User Interface of iLSB 


Let us demonstrate how iLSB scaffolds investigative learning process with an example of initial question 
“What is global warming?”. Learners are first expected to input “global warming” as q-keyword into a search 
engine and to gather Web resources related to “global warming”. Next, they are expected to store keywords 
such as “Greenhouse gas”, and “Kyoto Protocol” in the keyword repository while navigating the Web 
resources/pages, which represent the contents learned about the initial question. They are then expected to 
associate the keywords to construct their knowledge. After that, they are expected to find out sub q-keywords 
such as “Greenhouse gas” to be further investigated from the stored keywords, and to add the sub q-keywords 
to the question tree viewer. They are next expected to investigate these sub q-keywords in the same way. 
They are finally expected to create a question tree, whose root is “global warming”. 


2.4 Issue and Purpose 


Even with iLSB, some learners still have difficulties in creating their own scenario, which tends to be 
insufficient. It is necessary for such learners to do exercises to acquire skills in Web-based investigative 
learning. For exercises in Web-based investigative learning, it is an important issue how to define exercise 
questions. We have provided learners only with an initial question as exercise one so far. However, they 
often have difficulties in finding out sub-questions to be decomposed from the contents learned. In order to 
improve skills in question decomposition, we define an exercise question as the one including an initial 
question and related sub-question candidates, which expects learners to select proper sub-questions. 

In this paper, we propose a method for extracting keywords (called c-keywords) as q-keyword candidates 
to be included in an exercise question from LOD, which are related to an initial question. This method then 
generates an exercise question to be presented to learners, which is composed of the initial question and 
extracted c-keywords. This exercise question intends to improve learners’ skill for appropriate question 
decomposition. c-keywords are divided into positive c-keywords that have strong relationship with the initial 
question and negative c-keywords that have weak or less proper relationship with the initial question. In 
creating a learning scenario, learners are expected to select positive c-keywords and to avoid selecting 
negative ones. Such selection allows learners to enhance their skills in question decomposition. 


3. GENERATING EXERCISE QUESTIONS 


3.1 Linked Open Data (LOD) 


LOD is a set of structured data interlinking with related ones on the Web. In this work, we use DBpedia 
Japanese as LOD (DBpedia Japanese, 2016) whose data are extracted from Japanese Wikipedia. The data of 
DBpedia Japanese are stored in RDF format, which is described with triplet such as subject, predicate and 
object. A collection of RDF data is viewed as a network structure. These data are obtained using query 
language SPARQL. By sending SPARQL query to DBpedia Japanese, it is possible to obtain keywords 
(c-keywords) related to q-keyword representing an initial question. 
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Figure 2. Framework of Generating Exercise Question 
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3.2 Word2vec 


Word2vec is an algorithm that analyzes words in documents to generate word vectors including words and 
their adjacent words. It learns weights of adjacent relations between words by means of neural network from 
a large number of documents as input. 

In order to generate word vectors from Japanese Wikipedia, we use MeCab (Kudo, 2013) to extract words 
from the sentences, which are inputted to Word2vec. The extracted vectors are used to calculate cosine 
similarity between the initial question and c-keywords to be obtained from DBpedia Japanese. 


3.3 Framework of Exercise Questions 


Figure 2 shows the framework for generating an exercise question for Web-based investigative learning. In 
the proposed method, the exercise questions are generated by means of DBpedia Japanese and Word2vec, 
which include positive c-keywords highly related to the initial question and negative c-keywords less related 
to the initial question. 

First of all, this method sets up an initial question, and sends SPARQL queries for extracting c-keywords to 
DBpedia Japanese. As for all the extracted keywords, it calculates cosine similarity with q-keyword 
representing the initial question by means of word vectors extracted with Word2vec in advance. This method 
extracts keywords with higher similarity as positive c-keywords, and also extracts keywords with lower 
similarity as negative c-keywords. An exercise question is then generated. Learners are expected to investigate 
the initial question with iLSB and to obtain sub-questions from these c-keywords particularly from the positive 
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ones to create proper learning scenario. Such exercise question intends to help learners select sub q-keywords 
for scenario creation. In addition, it is possible to adjust difficulty of exercise question by changing the amount 
and ratio of positive and negative c-keywords to be provided to learners. Such adjustment allows learners to 
improve their skills in creating proper learning scenario. 


3.4 Extracting c-keywords with SPARQL Query 


Both positive and negative c-keywords are extracted with SPARQL query in consideration of the links with 
the initial question on DBpedia Japanese. Figure 3 shows the links between an initial question (“Global 
warming” as the q-keyword) and its positive/negative c-keywords on DBpedia Japanese. In Figure 3, positive 
c-keywords are expressed as yellow nodes, and negative c-keywords are expressed as red nodes. 

Since positive c-keywords should be highly related to the initial question, it is necessary to extract 
keywords closely related to the initial question on DBpedia Japanese. This method accordingly searches and 
extracts keywords such as Kyoto Protocol in Figure 3 that are bidirectionally linked with the initial question 
on DBpedia Japanese. The method moreover searches keywords such as Chlorofluorocarbon and Water 
vapor that are bidirectionally linked with the extracted positive c-keywords. However, the method does not 
extract the searched keywords such as Water Vapor that have no link to the initial question. 

On the other hand, negative c-keywords should not be far from the initial question so that learners could 
not immediately understand these are not necessary for question decomposition. This method accordingly 
searches keywords as negative ones like “Water vapor” that are bidirectionally linked with positive 
c-keywords and that have no link with the initial question. 


4. CASE STUDIES 


4.1 Appropriateness of c-keywords 


4.1.1 Purpose and Procedure 


We conducted a case study whose purpose was to ascertain whether c-keywords extracted by the proposed 
method were appropriate as positive and negative q-keywords as to initial question. In this study, three of the 
authors evaluated each c-keyword as “appropriate” or “inappropriate” by referring to reliable Web resources 
related to initial question. The appropriateness was decided by majority of three authors. In this study, we 
prepared two initial questions, “What is renewable energy?” and “What is foodborne illness?”. 

The appropriateness of positive c-keywords was calculated as the ratio of positive c-keywords evaluated 
as “appropriate” to all positive ones. The appropriateness of negative c-keywords was calculated as the ratio 
of negative c-keywords evaluated as “inappropriate” to all negative ones. 


4.1.2 Results 


Figure 4 shows the results of appropriateness of positive and negative c-keywords for each initial question 
and for total. As shown in this figure, “appropriate” ratings of both positive and negative c-keywords were 
about 80% for each initial question. This suggests that positive and negative c-keywords extracted by the 
proposed method are almost appropriate. 


4.1.3 Discussion 


Let us discuss inappropriate c-keywords. According to Figure 4, about 20% of c-keywords were considered 
as inappropriate keywords. Most of positive c-keywords evaluated as inappropriate represent extremely 
subdivided examples and company names. Although these c-keywords are highly related to the initial 
question on DBpedia Japanese, these are not appropriate as sub-questions for the initial questions. As for the 
negative c-keywords evaluated as inappropriate, there was a negative keyword mackerel that often causes 
food poisoning, for example. However, this is considered to be a negative c-keyword because there is no link 
to the initial question foodborne illness. How to remove these inappropriate c-keywords is one of our future 
work. 
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Figure 5. Procedure 


4.2 Effectiveness of Exercise Question 


4.2.1 Purpose and Procedure 


We conducted a case study whose purpose was to ascertain whether exercise questions generated were 
effective in creating appropriate learning scenario. Participants were 14 graduate and undergraduate students 
in science and technology. They conducted Web-based investigative learning twice only with initial question 
and with exercise question including initial question and c-keywords. The initial questions provided were 
“IQ1: What is renewable energy?” and “IQ2: What is foodborne illness?” IQ1 was represented as renewable 
energy, and IQ2 was also represented as foodborne illness. The procedure of this study is shown in Figure 5. 
The participants were divided into the following four groups in order to remove the order effects of 
presenting the initial questions and c-keywords: 


(a) Investigating IQ1, then investigating IQ2 with c-keywords, 
(b) Investigating IQ2 with c-keywords, then investigating IQ1, 
(c) Investigating IQ2, then investigating IQ1 with c-keywords, and 
(d) Investigating IQ1 with c-keywords, then investigating IQ2. 


The participants first practiced using iLSB for 30 minutes. Then, they conducted Web-based 
investigative learning for each initial question from 30 to 60 minutes with iLSB. In investigative learning 
with c-keywords, the list of c-keywords was provided on paper. The participants were asked to obtain 
sub-questions from the c-keywords to create learning scenario. 

After their investigative learning, three of the authors manually evaluated each question decomposition 
in their scenarios with three ratings, “appropriate”, “weak appropriate” and “inappropriate” by referring to 
reliable Web resources about the initial questions. The evaluation was decided by majority vote of three 
raters. When the three ratings did not match with each other, it was determined as “weak appropriate”. 

In order to ascertain the effectiveness of exercise questions, in addition, the participants were divided 
into high rating group and low rating group according to the ratio of “appropriate” ratings for question 
decomposition in learning scenario created without c-keywords. The high rating group had 6 participants. 
Three of them investigated IQ1, and the others investigated IQ2. The remaining 8 participants were classified 
into the low rating group. In evaluating the effectiveness of the exercise questions, we set the following 
hypothesis: 


Hypothesis: Presenting c-keywords to learners promotes appropriate question decomposition. 
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4.2.2 Results 


Figure 6 shows the ratio of ratings for question decomposition (QD for short) across all participants. In order 
to confirm whether the proposed method supports the hypothesis, we compared the ratio of ratings for QD in 
learning scenarios created with and without c-keywords. As for QD evaluated as “appropriate”, the ratio of 
ratings for QD with c-keywords was significantly higher than the one without c-keywords from the results of 
the one-sided t-test (t(26)=-2.34, p<.05). As for the ratios of QD evaluated as “weak appropriate” and 
“inappropriate”, on the other hand, the ratios of ratings for QD with c-keywords decreased compared to the 
ones without c-keywords. In particular, the ratio of QD evaluated as “weak appropriate” decreased 
significantly (t(26)=2.56, p<.01). 

Figure 7 shows the ratios of ratings for QD in the high rating group. From the results of one-sided t-test, 
there were no significant differences between with c-keywords and without c-keywords. These results 
suggest that the hypothesis is not supported in the high rating group. On the other hand, Figure 8 shows the 
ratios of ratings for QD in the low rating group. From the results of one-sided t-test, the ratio of “appropriate” 
ratings for QD with c-keywords significantly increased compared to the one without c-keywords (t(14)=-4.39, 
p<.001). In addition, the ratios of “weak appropriate” and “inappropriate” ratings for QD with c-keywords 
also significantly decreased (Weak appropriate: t(14)=2.75, p<.01, Inappropriate: t(14)=2.72, p<.01). From 
these results, the hypothesis is supported for the low rating group. 

Figure 9 shows the ratios of question decomposition rated as “inappropriate” and involving positive 
c-keywords evaluated as inappropriate in 4.1. From these results, about 60-65% of such question 
decomposition used inappropriate positive c-keywords in both groups. 
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4.2.3 Discussion 


Let us first discuss question decomposition conducted by all participants shown in Figure 6. Overall, 
presenting c-keywords significantly contributes to increasing the “appropriate” rating for QD and to 
decreasing the “weak appropriate” rating for QD. Although the “inappropriate” rating did not significantly 
decrease, the results suggest c-keywords could promote appropriate question decomposition. 

As for the high rating group, secondly, the results shown in Figure 7 suggest presentation of c-keywords 
was not effective for their question decomposition. In investigative learning without c-keywords, they could 
conduct appropriate question decomposition as shown in this Figure. This seems the main reason why 
c-keywords did not contribute. In addition, the ratio of the “inappropriate” rating for QD with c-keywords 
slightly increased. As shown in Figure 9, this suggests that inappropriate positive c-keywords prevented them 
from appropriate question decomposition. We accordingly need to improve the accuracy of extracting 
positive c-keywords by means of LOD and Word2vec. 

Let me next discuss about the low rating group. The results shown in Figure 8 suggest that presenting 
c-keywords promotes their appropriate question decomposition, and that exercise questions could be 
effective scaffolding for novice learners. In addition, the ratio of the “appropriate” rating for QD with 
c-keywords in Figure 8 was higher than the one for QD without c-keywords in the high rating group in 
Figure 7. This result suggests that c-keywords allow the low rating group to conduct appropriate 
decomposition in the same way as the high rating group. 


5. CONCLUSION 


This paper has addressed the issue of how to generate exercise questions for self-directed investigative 
learning on the Web. Toward this issue, we have proposed a method of extracting and selecting keywords 
related to an initial question as c-keywords by means of LOD and Word2vec to present as exercise question. 
We have also conducted case studies to ascertain the appropriateness and effectiveness of the proposed 
method. The results suggest that the proposed method provides appropriate exercise questions, and promotes 
appropriate question decomposition particularly for novice learners. On the other hand, we have found out 
the necessity of improving the accuracy of c-keywords. 

In future, we will address the issue how to generate exercise questions according to learners’ skills by 
changing the number and the ratio of positive and negative c-keywords to be included. 
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